Search for: All records

Egocentric and exocentric perspectives of human action differ significantly, yet overcoming this extreme viewpoint gap is critical in augmented reality and robotics. We propose VIEWPOINTROSETTA, an approach that unlocks large-scale unpaired ego and exo video data to learn clip-level viewpoint-invariant video representations. Our framework introduces (1) a diffusion-based Rosetta Stone Translator (RST), which, leveraging a moderate amount of synchronized multi-view videos, serves as a translator in feature space to decipher the alignment between unpaired ego and exo data, and (2) a dual encoder that aligns unpaired data representations through contrastive learning with RST-based synthetic feature augmentation and soft alignment. To evaluate the learned features in a standardized setting, we construct a new cross-view benchmark using Ego-Exo4D, covering cross-view retrieval, action recognition, and skill assessment tasks. Our framework demonstrates superior cross-view understanding compared to previous view-invariant learning and ego video representation learning approaches, and opens the door to bringing vast amounts of traditional third-person video to bear on the more nascent first-person setting. 

We investigate exocentric-to-egocentric cross-view translation, which aims to generate a first-person (egocentric) view of an actor based on a video recording that captures the actor from a third-person (exocentric) perspective. To this end, we propose a generative framework called Exo2Ego that decouples the translation process into two stages: high-level structure transformation, which explicitly encourages cross-view correspondence between exocentric and egocentric views, and a diffusion-based pixel-level hallucination, which incorporates a hand layout prior to enhance the fidelity of the generated egocentric view. To pave the way for future advancements in this field, we curate a comprehensive exo-to-ego cross-view translation benchmark. It consists of a diverse collection of synchronized ego-exo tabletop activity video pairs sourced from three public datasets: H2O, Aria Pilot, and Assembly101. The experimental results validate that Exo2Ego delivers photorealistic video results with clear hand manipulation details and outperforms several baselines in terms of both synthesis quality and generalization ability to new actions. 

During Expedition 386, two Giant Piston Corer (GPC) system deployments in the northern study area (Basin S3) of the southern Japan Trench (Figure F1) resulted in the recovery of cores from four holes at Site M0091 (Figure F2). The water depth was between 7802 and 7812 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Site M0091 were cored during Week 6 of the offshore phase. In total, 51.94 m of cores (Table T1) and 53.5 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired, respectively, in the focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, one Giant Piston Corer (GPC) system deployment at Basin C/N1 in the boundary area between the central and northern Japan Trench (Figure F1) resulted in the recovery of cores from two holes at Site M0093 (Figure F2). The water depth was 7454 m below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Site M0093 were cored during Week 7 of the offshore phase. In total, 26.91 m of cores (Table T1) and 3.89 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired, respectively, in this focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, one Giant Piston Corer (GPC) system deployment at the boundary area between the central and northern Japan Trench (Figure F1) resulted in the recovery of cores from two holes at Site M0094 (Figure F2). The water depth was 7469 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary materials) due to decisions to move between sites based on weather and current conditions. Holes at Site M0094 were acquired during Week 7 of the offshore phase. In total, 19.065 m of cores (Table T1) and 5.8 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired in this focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary materials). 

During Expedition 386, a total of five Giant Piston Corer (GPC) system deployments in the central Japan Trench (Basin C2; Figure F1) resulted in the recovery of cores from six holes at Site M0083 and four at Site M0089 (Figure F2). The water depth ranged 7602–7626 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Sites M0083 and M0089 were cored during Weeks 2–4 of the offshore phase. In this focus area, a total of 154 m of cores (Table T1) were recovered. In addition, 121 km of hydroacoustic profiles (see Hydroacoustics) were acquired. Further operations details, including winch log and inclinometer information for all sites, are found in Coring methodology in the Expedition 386 methods chapter (Strasser, 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, a total of five Giant Piston Corer (GPC) system deployments in the northern Japan Trench (Basin N3; Figure F1) resulted in the recovery of cores from six holes at Site M0084 and four at Site M0085 (Figure F2). The water depth was between 7590 and 7603 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Cores from Sites M0084 and M0085 were acquired during Weeks 2, 3, and 5 of the offshore phase. In total, 149.2 m of cores (Table T1) and 133 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired, respectively, in this focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and in the associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, two Giant Piston Corer (GPC) system deployments at this study area in the northern Japan Trench (Basin N2; Figure F1) resulted in the recovery of cores from four holes at Site M0088 (Figure F2). The water depth was between 7525 and 7550 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Site M0088 were cored during Week 4 of the offshore phase. In total, 56.205 m of cores (Table T1) and 49.7 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired in this focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). Note that inclinometer data were not properly recorded and are therefore not reported for Site M0088.